A comparison of qSOFA, SIRS and NEWS in predicting the accuracy of mortality in patients with suspected sepsis: A meta-analysis

Objective To identify and compare prognostic accuracy of quick Sequential Organ Failure Assessment (qSOFA) score, Systemic Inflammatory Response Syndrome (SIRS) criteria, and National Early Warning Score (NEWS) to predict mortality in patients with suspected sepsis. Methods This meta-analysis followed accordance with the recommendations of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement. We searched PubMed, EMBASE, Web of Science, and the Cochrane Library databases from establishment of the database to November 29, 2021. The pooled sensitivity and specificity with 95% CIs were calculated using a bivariate random-effects model (BRM). Hierarchical summary receiver operating characteristic (HSROC) curves were generated to assess the overall prognostic accuracy. Results Data of 62338 patients from 26 studies were included in this meta-analysis. qSOFA had the highest specificity and the lowest sensitivity with a specificity of 0.82 (95% CI: 0.76–0.86) and a sensitivity of 0.46 (95% CI: 0.39–0.53). SIRS had the highest sensitivity and the lowest specificity with a sensitivity of 0.82 (95% CI: 0.78–0.85) and a specificity 0.24 (95% CI: 0.19–0.29). NEWS had both an intermediate sensitivity and specificity with a sensitivity of 0.73 (95% CI: 0.63–0.81) and a specificity 0.52 (95% CI: 0.39–0.65). qSOFA showed higher overall prognostic accuracy than SIRS and NEWS by comparing HSROC curves. Conclusions Among qSOFA, SIRS and NEWS, qSOFA showed higher overall prognostic accuracy than SIRS and NEWS. However, no scoring system has both high sensitivity and specificity for predicting the accuracy of mortality in patients with suspected sepsis.


Introduction
In the United States, the sepsis incidence of 5.9% among hospitalized patients, a trend that has been increasing annually [1]. Although the mortality of sepsis has decreased in recent years, it is still the main cause of mortality worldwide [2]. The key strategies for a successful outcome in patients with sepsis are early recognition and timely therapy. However, accurate identification of sepsis is still a problem for clinicians. A reliable method to evaluate sepsis can help clinicians correctly identify sepsis, improve the initial treatment plan of patients, and ultimately improve the survival rate.
Sepsis-1 [3] in 1991 and Sepsis-2 [4] in 2001 suggest that sepsis should be defined as infection with Systemic Inflammatory Response Syndrome (SIRS). The definition of Sepsis-3 [5] was released in 2016 and recommended "Sequential Organ Failure Assessment (SOFA)" or "quick Sequential Organ Failure Assessment (qSOFA)". The qSOFA score is a simplified score based on the SOFA score, which is said to be more accurate than SOFA in departments outside the intensive care unit (ICU) [6]. In recent years, some studies have tried to evaluate the prognostic accuracy of qSOFA and SIRS. In general, SIRS has high sensitivity but low specificity, and qSOFA has high specificity but low sensitivity in the prognosis of sepsis [7]. In addition, the National Early Warning Score (NEWS) is widely used in the UK as a tool to assess and monitor the clinical status of hospital patients and has the same or higher prognostic accuracy [8]. However, it is unclear which scoring system has higher prognostic accuracy in patients with suspected sepsis.
We included all studies that compared qSOFA, SIRS, and NEWS in suspected sepsis patients and performed a meta-analysis of the available studies to determine the accuracy of these scoring systems in predicting mortality in suspected sepsis patients.

Method
This meta-analysis was conducted in accordance with the recommendations of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement. The protocol for this review was prospectively registered in INPLASY (number INPLASY202140029) outside ICU, (2) the purpose was to evaluate or compare the accuracy of qSOFA, SIRS, and NEWS in predicting mortality, (3) a 2 × 2 contingency

Data extraction
Two investigators (Can Wang and Yuerong Zeng) independently extracted data from the selected articles. The extracted data were as follows: study characteristics (author, year of publication, country of origin, type of study), patient characteristics (selection criteria of patients, number of patients enrolled, age, sex, setting in which patient was seen), and outcomes (type of measured mortality, cut-off value of qSOFA, SIRS, and NEWS, TP, FP, FN, TN, sensitivity, specificity)

Study quality assessment
Two investigators (Can Wang and Yuerong Zeng) independently evaluated the quality of included studies according to the Quality Assessment of Diagnostic Accuracy Studies 2 (QUA-DAS-2) [9]. In case of any disagreement, the study group should discuss and resolve it. The evaluation content mainly consists of four parts: patient selection, index test, reference standard, flow and timing. According to the "yes", "no", or "uncertain" answers to the relevant landmark questions included in each part, the risk of bias can be judged as "low", "high", or "uncertain".

Statistical analysis
Statistical analysis was conducted using STATA15 and RevMan5.3. The pooled sensitivity and specificity with 95% CIs were calculated using a bivariate random-effects model (BRM) [10]. Hierarchical summary receiver operating characteristic (HSROC) curves were generated to assess the overall prognostic accuracy. Lambda, Theta, and Beta are the estimated parameters of HSROC. Lambda is the natural logarithm of the diagnostic odds ratio (DOR), Theta is the mean of the log of sensitivity and the log of 1-specificity, Beta is the parameter that defines the shape of the summary curve [11]. The Beta value significant difference from zero indicates that the curve is asymmetric, which is not suitable for calculating the pooled value of accuracy [11]. A beta value equal to or close to zero indicates that the curve is symmetrical, and lambda can be used to evaluate the overall prediction accuracy [11]. The post-test probability was assessed by Fagan's nomogram.
The heterogeneity was evaluated by the I 2 test. If I 2 � 50%, P � 0.1, the heterogeneity among the studies was acceptable; If I 2 > 50%, P < 0.1, the heterogeneity among the studies is significant, and the source of heterogeneity should be analyzed. The threshold effect was not judged because the Stata Midas module was used for statistical analysis. Univariate metaregression and subgroup analysis were used to explore the source of heterogeneity. Deek' s funnel plot was made to assess the publication bias.

Quality assessment of studies
The individual and overall quality assessment results of the 26 included studies were shown in S1 Fig. Overall, the included studies showed that there were risks in two of the four areas.
Except for the study of Brink, et al [14], all other studies included consecutive or random cases, avoiding inappropriate exclusion. Because the cut-off values of the 11 studies [8,16,17,25,26,28,[32][33][34][35][36] were not determined in advance, there were high risks in the index test. Dick's funnel plot suggested that there was no publication bias (Fig 2).  Table 2. Beta was 0.12 (Z = -0.74, P = 0.459), which indicates that the curve is symmetric. Lambda was 1.23 (95% CI: 0.91-1.55), which represents corresponding the DOR of 3.79. Fagan' s nomogram ( Fig 4A) showed that qSOFA increased the possibility of mortality in patients with suspected sepsis to 39% when the pretest probability of mortality was 20%.

Data synthesis and meta-analysis
Predictive validity of SIRS. The pooled sensitivity and specificity of SIRS for predicting mortality in suspected sepsis patients were 0.82 (95% CI: 0.78-0.85) and 0.24 (95% CI: 0.19-0.29), respectively (S3 Fig). The HSROC curve and the estimated parameters of SIRS were shown in Fig 3B and Table 2. Significant heterogeneity between studies can be observed in the combined results of sensitivity and specificity (I 2 = 77.08% and I 2 = 98.32%). Meta-regression analysis and subgroup analysis showed that the heterogeneity of sensitivity was caused by the types of studies, patients' settings, the types of patients, and the types of mortality measured, and the heterogeneity of specificity was caused by the types of study and types of patients (S5B  Table 2). Fagan's nomogram ( Fig 4C) showed that NEWS increased the possibility of mortality in patients with suspected sepsis to 28% when the pretest probability of mortality was 20%.
Performance comparison of qSOFA, SIRS and NEWS. The performance of the qSOFA, SIRS and NEWS in predicting mortality in patients with suspected sepsis were presented in Fig 5 and Table 3. In direct comparisons, qSOFA showed higher overall prediction accuracy than SIRS and NEWS with the diagnostic odds ratio (DOR). In addition, the AUC in qSOFA and NEWS were higher than in SIRS. qSOFA provided the highest specificity for predicting mortality followed by NEWS and SIRS. On the contrary, the SIRS provided the highest sensitivity for predicting mortality followed by NEWS and qSOFA. qSOFA showed better post-test probability than SIRS and NEWS, representing patients with suspected sepsis met qSOFA the greater chance of die.

Discussion
This is the first meta-analysis comparing the prognostic accuracy of qSOFA, SIRS, and NEWS to predict mortality in patients with suspected sepsis. Our meta-analysis identified 26 clinical studies, including 62338 patients with suspected sepsis. However, no scoring system has both high sensitivity and specificity for predicting the accuracy of mortality in patients with suspected sepsis. These scoring systems have their advantages and disadvantages. qSOFA is a bedside assessment tool recommended by the Third International Sepsis Consensus Definitions Task Force used to assess patients with suspected sepsis outside the ICU [5]. Our findings suggest that qSOFA shows the highest overall prediction accuracy of mortality and has high specificity. Therefore, qSOFA is of great value in predicting the mortality of patients with suspected sepsis. It can more accurately identify patients with a high risk of death than the other two scoring systems. However, our results show that qSOFA has low sensitivity, which means that false negative is high and it is easy to miss or delay treatment. The reasons for the low sensitivity of qSOFA may be as follows. On the one hand, the supporting paper [6] derived and tested among critically ill patients and there will be deviation and sensitivity will be decreased for the general patient population. On the other hand, qSOFA � 2 is too strictly and patients will be in a late disease state with a worse prognosis using it [37]. Due to the lethality of sepsis, a screening mechanism showing high sensitivity is needed [38]. Some studies have proposed that reducing the qSOFA cut-off to 1 or combined with lactate levels can improve the sensitivity [39].
It is universally acknowledged that any scoring system used to determine sepsis should tend to be higher sensitivity rather than specificity because the cost of delaying or missing treatment caused by false negatives is much greater than the cost of unnecessary antibiotics caused by false positives [13]. We found that SIRS had the highest sensitivity among the three scoring systems. However, SIRS is too poor to predict mortality in patients with suspected sepsis. Whether direct comparison or indirect comparison, SIRS has the lowest prediction ability. SIRS is more suitable as a screening tool for early care and the prevention of missed diagnoses.

PLOS ONE
qSOFA, SIRS and NEWS predict mortality in sepsis NEWS is widely recommended to identify patients at risk of deterioration, which was launched by the Royal College of Physicians (RCP) in 2012 [40]. NEWS has the strongest ability to identify patients at risk of deterioration compared with other Early Warning Score

PLOS ONE
qSOFA, SIRS and NEWS predict mortality in sepsis (EWS) [41]. Meanwhile, more and more studies have proved that NEWS is a promising scoring system and can be used as an alternative screening tool for patients with suspected sepsis. In this meta-analysis of patients with suspected sepsis, we found that NEWS is slightly worse than qSOFA in terms of overall prediction ability, but avoids the extremely low sensitivity of qSOFA and has both an intermediate sensitivity and specificity. The strength of the NEWS is that it can be calculated based on physiological parameters alone, which is easier to implement than the other two scoring systems [42].
Our research also has some limitations. On the one hand, there is significant heterogeneity in our meta-analysis. The included studies were different types of studies (prospective or retrospective), different outcome indicators (in-hospital mortality or 30 / 28 / 60-day mortality), and different types of patients (sepsis or suspected sepsis). The above points are the sources of heterogeneity. On the other hand, the cut-off values of the included studies are different, and the cut-off values of some studies are not determined in advance. A predefined cut-off value help to reduce the sensitivity and specificity bias that may result from this data-driven method [43].

Conclusion
In conclusion, our results indicate that qSOFA showed higher overall prediction accuracy of mortality than SIRS and NEWS. The three scoring systems have limitations as a tool for predicting mortality in patients with suspected sepsis. A scoring system with both high sensitivity and specificity needs to be studied in the future.